Shifting wearout of storage disks

ABSTRACT

Technical solutions are described to forestall data loss caused by wearout of storage disks in an array of storage disks in a storage system by monitoring a rate of writes for a first storage disk in the array and determining a mean time to failure of the first storage disk. A start time is determined based on the mean time to failure, a number of storage disks in the array, and a time to replace a storage disk in the array. At the start time, a notification is issued as an alert to replace the first storage disk to forestall data loss caused by wearout of a second storage disk in conjunction with a wearout of the first storage disk.

BACKGROUND

The present application relates generally to storage systems, and morespecifically, to storage systems that include multiple storage disks.

Storage systems include multiple disks arranged in an array, such asusing Redundant Array of Independent Disks (RAID) techniques. Typically,such arrangements are effective for Hard-Disk Drive (HDD) dataprotection because the probability of encountering a secondary faultduring repair of a first failed device is extremely low. Even though atfirst glance one might assume that a Solid State Drive (SSD) will havesimilar failure characteristics as of an HDD, that is not the case.Attempting to apply RAID techniques to SSDs can prove ineffective sinceSSD failure more often occurs from wearout and not due to normal randomfailure mechanisms. Further, SSD wearout failures typically occur in avery small time window, which may cause all the SSD devices in the RAIDarray to fail at about the same time.

SUMMARY

According to an embodiment, a method for forestalling data loss causedby wearout of storage disks in an array of storage disks in a storagesystem includes monitoring, by a controller, a rate of writes for afirst storage disk in the array. The method also includes determining,by the controller, a predicted mean time to failure of the first storagedisk based on the rate of writes. The method also includes determining,by the controller, a start time based on the predicted mean time tofailure, a number of storage disks in the array, and an estimated timetaken to replace a storage disk in the array. The method also includesissuing, by the controller, at the start time, a notification. Thenotification includes an alert to replace a first storage disk, wherethe notification is issued prior to the first storage disk reaching awearout, and where replacing the first storage disk in response to thenotification forestalls data loss caused by a wearout of a secondstorage disk in conjunction with the wearout of the first storagedevice.

According to another embodiment, a storage system includes a pluralityof storage disks in an array, and a controller that stores data acrossthe storage disks, where the controller forestalls data loss caused bywearout of storage disks in the array of storage disks. The controllermonitors a rate of writes for a first storage disk in the array. Thestorage system also includes determine a mean time to failure of thefirst storage disk based on the rate of write. The controller alsodetermines a start time based on the mean time to failure, a number ofstorage disks in the array, and a time to replace a storage disk in thearray. The controller also issues at the start time, a notification, thenotification includes an alert to replace the first storage disk, wherereplacement of the first storage disk at the start time forestalls thedata loss caused by a wearout of a second storage disk of the array inconjunction with a wearout of the first storage disk.

According to another embodiment, a computer product for forestallingdata loss caused by wearout of storage disks in an array of storagedisks in a storage system includes computer readable storage medium,which includes computer executable instructions. The computer executableinstructions cause the processor to monitor a rate of writes for a firststorage disk in the array. The computer executable instructions alsocause the processor to determine a mean time to failure of the firststorage disk based on the rate of write. The computer executableinstructions also cause the processor to determine a start time based onthe mean time to failure, a number of storage disks in the array, and atime to replace a storage disk in the array. The computer executableinstructions also cause the processor to issue at the start time, anotification, the notification includes an alert to replace the firststorage disk, where replacement of the first storage disk at the starttime forestalls the data loss caused by a wearout of a second storagedisk of the array in conjunction with a wearout of the first storagedisk.

BRIEF DESCRIPTION OF THE DRAWINGS

The examples described throughout the present document may be betterunderstood with reference to the following drawings and description. Thecomponents in the figures are not necessarily to scale. Moreover, in thefigures, like-referenced numerals designate corresponding partsthroughout the different views.

FIG. 1 illustrates failure modes that fall into the different categoriesdepicted by the bathtub curve.

FIG. 2 illustrates an example storage system in accordance with anembodiment.

FIG. 3 illustrates a unit normal plot for a normal distributionhighlighting the probability of occurrence of a second storage diskfailing in conjunction with a first storage disk.

FIG. 4 illustrates an example failure model of a storage disk, accordingto an embodiment.

FIG. 5 illustrates example probability of storage disk failure,according to an embodiment.

FIG. 6 illustrates example conversion of probability of storage diskfailure from number of write to time to failure, according to anembodiment.

FIG. 7 illustrates a flowchart of example logic to reduce the occurrenceof data loss in the storage system in accordance with an embodiment.

FIG. 8 illustrates an example schedule for staggering checksum acrossstorage disks to vary time to wearout in accordance with an embodiment.

FIG. 9 illustrates a flowchart of example logic to generate a staggerschedule to reduce the occurrence of data loss in the storage system inaccordance with an embodiment.

FIG. 10 illustrates an example schedule for staggering checksum acrossstorage disks to vary wearout in accordance with an embodiment.

FIG. 11 illustrates an example storage system in accordance with anembodiment.

FIG. 12 illustrates a flowchart of example logic to use one or morereplacement disks to reduce the occurrence of data loss in the storagesystem in accordance with an embodiment.

FIG. 13 illustrates an example schedule of using 1 spare disk to shiftwearout 8 weeks in accordance with an embodiment.

FIG. 14 illustrates an example schedule of using 1 spare disk to shiftwearout 2 weeks in accordance with an embodiment.

FIG. 15 illustrates a flowchart of example logic to use two replacementdisks to reduce the occurrence of data loss in the storage system inaccordance with an embodiment.

FIG. 16 illustrates an example schedule of using 2 spare disks to shiftwearout in 8 weeks increments in accordance with an embodiment.

FIG. 17 illustrates an example schedule of utilizing spare storage diskto shift writes across an array of storage disks to vary wearout inaccordance with an embodiment.

FIG. 18 illustrates example probability of storage disk failure withprior alerting.

FIG. 19 illustrates a flowchart of example logic to determine alerttimes to reduce the occurrence of data loss in the storage system inaccordance with an embodiment.

DETAILED DESCRIPTION

Disclosed here are technical solutions for improving reliability of astorage system by staggering wearouts of storage disks that the storagesystem uses. The technical solutions facilitate the storage system touse storage disks that are susceptible to wearout failures, such asFlash-based storage disks, by reducing probability of a data loss. Forexample, the storage system staggers the wearout such that whilereplacing a first storage disk that may have failed, a probability of asecond storage disk failing during the replacement period is reduced.

Reliability Engineering is the art and branch of science concerned withpredicting the reliability of devices and systems and with thedevelopment of new designs to make them more reliable. This includestechniques to enable systems and devices to remain operational in thepresence of a component failure. Common terminology includes:

Time to Failure=T Failure Rate=F(t)=P (T<t) (CDF—Cumulative FailureRate) Reliability=R(t)=P (T>t)=1−F(t)

Hazard Rate=h(t)=P (T<t+dt|T>t) (Conditional Probability of Failure,where dt is an incremental change in t)

MTTF=Mean Time To Failure

MTBF=Mean Time Between Failure (Used to Model with Repair)

Reliability Engineering recognizes three common failure modes that fallinto the different categories depicted by the bathtub curve in FIG. 1.The first failure mode encountered during the life of a system or deviceis known as “early life” failure, which typically occurs in the firstfew months after the device is placed into use. Early life failuresexhibit a high initial hazard rate that decreases over time, eventuallydisappearing all together. Early life failures result primarily frommanufacturing process-induced defects. Manufacturing processes thatproduce sophisticated devices such as computer system rely on complexsteps and procedures that are regulated. Despite rigorous control suchas pressurized clean rooms, high purity chemicals, highly calibratedequipment, well trained operators, some imperfections and deviation fromspecification are observed in a subset of the devices produced. Thesemanufacturing induced defects and flaws do not always result in devicesthat do not work. Rather, such defect manifest themselves in ways wherea device initially appears to operate properly, but fails after a firstfew operations or where operational tolerance drifts causingintermittent system failure. Temperature cycling and mechanicalvibration can be catalysts cause early life failures to surface and evenaccelerate the time to failure. Early life failures are often modeledwith a Weibull failure rate distribution with a Beta less than 1.

The second and more prevalent failure mode is known as “useful life”failures. These are random failures that occur at a constant hazard ratefor most of the useful life of the device. The useful life failure modefollows an exponential distribution, which is a special case of theWeibull distribution where Beta=1. The exponential failure rate has nomemory since the hazard rate, the conditional probability of failure inthe next delta t time period remains constant throughout the useful lifeof the device.

The third failure mode is known as “wearout,” which has an increasinghazard rate and is a common failure mode for moving mechanical partssubject to friction and wear. For example, electrical devices areretired, or replaced with newer devices, before they wearout. Wearout isa recognized failure mode for devices such as incandescent light bulbs,windshield wipers or car tires where the failure rate abruptly increasesafter some period of use. Car tires, for example, are more likely tofail after a certain number of miles of driving. As the tread begins towear, the tires become more susceptible to failure, such as puncture.Wearout has an increasing failure rate and can be modeled with a Weibulldistribution where Beta is greater than 1, or with a Normal Failure RateDistribution.

Storage disks, such as Hard Disk Drives (HDDs), Solid State Drives(SDDs), Flash memory, or any other type of storage disks, are complexelectro-mechanical devices that can exhibit wearout. Typically, astorage disk exhibits a long period of useful life before encounteringwearout. For example, most HDD failures occur during the useful life,the failures follow an exponential failure rate and have a Mean Time ToFailure (MTTF) that is relatively high. Clusters of HDD failureoccurring in a short time window are unlikely unless there is some typeof systemic problem with the devices.

For example assume a population of high quality, well-built HDDs with aconstant hazard rate=0.005. Ignoring the brief early life failure periodand assuming that the HDD is retired before it reaches wearout, thereliability of this population can be modeled with an exponentialdistribution. The MTTF is the inverse of the constant hazard rate of0.005 which is 200 years or 200 device-years when measuring a populationof HDDs. That is, with a population of exactly 200 HDDs run continuouslyfor 1 year, one would expect on average a single (1) HDD to fail duringthat time. For 400 HDDs it would be 2 failures.

Nevertheless, even with a relatively low failure rate of HDDs, theimplications of a single device becoming nonfunctional is catastrophicsince the data on the device is lost when it fails. With large systemenvironments, running thousands if not tens of thousands of drives,failures are more common, as evinced by calculations described earlier.Accordingly, for example, if a population of 1000 HDDs is run for ayear, on average 5 HDD failures are expected, and for 10,000 HDDs 50 HDDfailures are expected. Thus, with a MTTF of 200 device-years for a largepopulation of HDDs, device failure is estimated to be relative common.

Hence, storage systems, such as a storage system 200 illustrated in FIG.2 uses several storage disks to store data. The storage system 200includes a controller 210 that stores the data across the storage disks.The storage disks are in an array of storage disks 220.

The controller 210 is hardware that uses software such as firmware anddrivers to implement a scheme to store the data across the array ofstorage disks 220. The controller 210 may be a processor. The controller210 may be a Serial Advanced Technology Attachment (SATA) or a ParallelAdvanced Technology Attachment (PATA) controller that implements theRAID scheme. Alternatively or in addition, the controller 210 maycommunicate with the array of storage disks 220 via a bus, such as aninterface or a connector. The bus may be a SATA connector, a PATAconnector, a Peripheral Component Interconnect (PCI) Express connector,a Small Computer System Interface (SCSI) connector, a M.2 connector, orany other type of interface to transfer data back and forth to and fromthe array of storage disks 220.

The array of the storage disks 220 includes one or more non-volatilestorage disks that store computer readable data. The array of storagedisks 220 may include any number of storage disks, such as one, two,three, five, six, and so on. A storage disk in the array may be a HDD, aSDD, a Flash memory, or any other type of non-volatile memory. The arrayof storage disks 220 may include a combination of storage disks of theabove type. The array of storage disks 220 store symbols or elements. Asymbol or an element is a fundamental unit of data or checksum. ForHDDs, symbols are typically held in sets of sequential sectors. Anelement is composed of a fixed number of bytes. It is also common todefine elements as a fixed number of blocks. A block thus, represents afixed number of bytes. A stripe is a complete and connected set of dataelements and checksum elements that are dependently related to thechecksum computation relations, which is calculated across thisdependent set of data elements. In coding theory, the stripe is the codeword or code instance. A strip is a segment of the stripe that resideson one physical storage disk. A strip is a collection of contiguouselements on a single storage disk. A strip contains data elements orchecksum elements or both, and from the same disk and stripe. The termstrip and column may be used interchangeably. In coding theory the stripis associated with the code word and is sometime called the stripe unit.The set of strips in a code word form a stripe. Typically, stripscontain the same number of elements. In some cases stripes are groupedtogether to form a higher level construct known as a stride.

For example, since storage disk failure results in data loss, thecontroller 210 uses redundancy techniques to prevent, or at leastmitigate the data loss. For example, the controller 210 may usetechniques, such as checksum, to maintain integrity of the data storedin the storage system 200. In an example, such as RAID-5, the checksumis parity, while in another example, such as RAID-6 the checksum isReed-Solomn code derived from a complex polynomial. Parity may indicatea count of particular elements in the data, for example a number of ‘1’s in binary data being stored. Reed-Solomon checksum is explainedfurther. For example, the controller 210 may implement a RAID scheme forthe array of storage disks 220. In case of a RAID scheme, the storagedisks in the array 210 store one or more instances of a RAID erasurecode is implemented. The erasure code includes one or more symbols. RAIDsystems improve performance and/or increase the availability of diskstorage systems. Typically, RAID distributes data across severalindependent storage disks. RAID may be deployed using one of severalschemes, each scheme having different characteristics, pros, and cons,associated with the scheme. Performance, availability, andutilization/efficiency (the percentage of the disks that hold data) aresome of the metrics used to measure effectiveness of the different RAIDschemes. There are, typically, tradeoffs associated with the RAIDschemes, because improvements in one attribute results in degradation inanother.

For example, RAID-0 includes striping of data across multiple storagedisks to improve performance. RAID-1 includes mirroring of data, keepingtwo exact copies of the data on two different storage disks to improveavailability and prevent data loss. The controller 210 may use more thanone RAID schemes together to gain combined benefits. For example,RAID-10 includes both data striping and mirroring across the storagedisks in the array 220. RAID-10 improves both performance andavailability.

The controller 210, in other examples, may implement a RAID-3, RAID-4,or a RAID-5 scheme, which use a single exclusive-or (XOR) check sum tocorrect for a single data element error. For example, in RAID-3 thecontroller 210 may use a byte level striping with a dedicated paritystorage disk. In RAID-4, the controller 210 uses a block level stripingwith a dedicated parity storage disk. In RAID-5, the controller 210 usesblock level striping like RAID-4, with distributed parity. That is, inRAID-5, there is no dedicated parity storage disk, rather, thecontroller 210 distributes the parity substantially uniformly across allthe disks in the array of storage disks 210. Thus, the controller 210,using RAID-5, reduces a dedicated parity storage disk from being as aperformance bottle neck. Using RAID-3, RAID-4, or RAID-5, the controller210 is capable of correcting a single data element fault when thelocation of the fault can be pinpointed. The controller 210 may correctfor a complete storage disk failure in any of the RAID-3, RAID-4, andRAID-5 schemes.

In another example, the controller 210 implements a RAID-6 scheme.RAID-6 includes block or Byte level striping with dual checksums. RAID-6facilitates correction of up to 2 data element faults when the faultscan be pinpointed. RAID-6 further facilitates pinpointing and correctinga single failure when the location of the failure is not known. RAID-6is based on Reed-Solomon (RS) error correction codes. RS-codes arenon-binary cyclic codes applied to data elements or symbols. Forexample, in an RS(n, k), k=the number of data symbols, n=the totalnumber of symbols, if m=symbol length in bits, then 0<k<n<2̂(m+1).Further, if c=symbol error correcting capability when the location ofthe error is not determined, then n−k=2c. In other words, the differencein the total number of symbols and the number of data symbols isdirectly proportional to the data correcting capability of the RS-code.The minimum distance is the number of bit differences between valid codewords. RS-code, thus facilitate a largest possible minimum distance forany linear code.

For non-binary codes, the minimum distance analogous to the Hammingdistance is given by d_(min)=n−k−1. Accordingly, the ability to correctfor faults when the fault location is determined independently, measureas e=erasure correcting capability is calculated by e=d_(min)−1=n−k=2c.That is, only half as many faults can be corrected when the location ofthe error is not determined independently. In such cases, using RS-codehelps to determine the location and subsequently correct the error. TheRS-codes, in general, is a series of simultaneous equations that areused to solve for unknowns. In the case of the storage system 200, theunknowns are either data symbols or the location of the symbols with thefault. RAID-6, for example, uses 2 equations to generate 2 independentchecksums, which are applied to each data elements in each row). TheRS-checksum equation can be expressed in polynomial form asQ(x)=d0+d1*x+d2*x̂2+ . . . d(N−1)*x̂(N−1), where Q(x)=Reed-Solomonchecksum where x=a; P(x)=Reed-Solomon checksum where x=1; and d0, d1 . .. dN=polynomial coefficients. The equation can be used to solve for acoefficients, to correct a data element or if the location of the faultis unknown it can be used to solve for the power of x to pinpoint thelocation.

The RAID3, RAID-4, and RAID-5 simple XOR is a special case of the RAID-6Reed-Solomon based scheme, of the polynomial sum where x=1 so that theequations then becomes P(x)=d0+d1+d2+ . . . d(N−1).

Different examples may use different variations of the general equationdescribed herein. A variation of the equation is typically a primitivepolynomial, that is the variation is analogous to a prime number whereit has no common roots. That ensures the solution always map to uniquevalues in a finite field, such as a Galois field.

For example, consider a storage system 200 using an array 210 thatincludes ten (10) HDDs, and in which the controller 210 implementsrecovery techniques to recover failure of a storage disk by accessingdata on other storage devices. For example, the storage system 200 usesa RAID-5 scheme over the array of ten disks. In RAID-5, when an HDDfailure is encountered the probability of a second HDD device failing inthe array 220 during the repair and rebuild time period is very low. Ifa second failure is encountered, before the repair and rebuild of thefirst failed drive is completed, data on both HDD devices will be lost.

Again, consider an HDD with a MTTF of 200 years further assuming repairof the first failed disk in the RAID-5 array of 10 disks takes up to 1week. The repair time may include time to detect the failure, order areplacement disk, receive the replacement disk, schedule the repair,replace the failed device and rebuild the data on the replacement disk.The probability of a second HDD failing of the 9 HDDs in the array thatare still functional is 0.00087 (0.087%) which can be calculated byadding the hazard rate of the 8 HDDs together or from F(t)̂8 since thestate of the other 7 HDDs is immaterial to the outcome.

Therefore, on average a second HDD failure will be encountered for every1000 repair attempts (0.87 is about equal to 1 so a roughly 1 per 1000).Based on these calculations, in an array of 1000 HDDs, over a year, thestorage system 200 expects five (5) HDD failures on average, since theprobability of encountering a second HDD failure in the arrays with theproblem during the 5 repairs is 0.0043 (0.43%).

In another example, if the controller 210 uses the RAID-6 scheme inwhich three (3) disks need to fail in the array before data loss isencountered, the probability drops to 0.0000046 (0.00046%). Thus, onaverage, the storage system 200 expects roughly 1 repair attempt and upto 3 disk fails for roughly every 200,000 repair attempts. If the HDDfailure rate is higher or the number of HDDs in the population ishigher, the probability of data loss increases, but for the range oftypical failure rates, even with very large numbers of devices, RAIDtechniques are highly effective at preventing data loss.

Now consider a case in which, the array of storage disks 220 includesSSDs (instead of HDDs). SSDs have different failure characteristics toHDDs because of which, applying data storage techniques, such as theRAID techniques to SSDs may be ineffective in comparison to HDDs. SSDfailures, typically, occur from wearout and not due to normal randomfailure mechanisms. Further, SSD wearout failures typically occur in avery small window where all the SSD devices in the array fail at aboutthe same time.

Unlike HDDs, wearout of SSDs is typically a consequence of normal use.This is because the maximum number of writes that can be made to an HDDbefore it exhibits wearout is typically on the order of a thousandtrillion or 10¹² writes. With SSDs the number of writes is on the orderof a trillion times lower dropping to between hundreds of thousands 10⁵to thousands 10³ depending on whether single level cell (SLC),multilevel cell (MLC), or triple level cell (TLC), NAND flash devicesare employed in the SSDs. There is ongoing engineering challenge tooptimize characteristics of SSDs by balancing density, performance, andendurance measured in maximum number of writes and data retention timeunderway. Thus unlike HDDs, random hardware failure of SSDs is typicallynot the primary failure mode. Device wearout can be prevalent when themaximum allowable media writes are exceeded. High input/outputoperations per second (TOP) applications, especially those with a highnumber of writes, are thus, more susceptible to SSD wearout.

Further, SSDs employ sophisticated wear leveling techniques that spreadwrites evenly across all the NAND Flash devices, and thus prevent themaximum number of writes at any one address of the NAND flash to beexceeded before the others address. This enables the device to beuseable for as long as possible. A down side of this technique is thatwhen the SSDs are used in a RAID array it is highly likely the deviceswill all fail at approximately the same time since between the wearleveling and protection schemes like RAID-5, the storage elementsreceive about the same number of writes over time.

For example, in the RAID-5 case described earlier, where data lossresults when two storage disks fail, using an array of SSDs results in arelatively smaller window of time to replace a failed disk. Just like incase of HDDs, after the first SSD failure the storage system 200 willtake a period of time to procure and replace the failed SSD. Once thefailed device is replaced a rebuild needs to occur whereby all the dataoff the remaining devices is read and XORed together to regenerate thedata on the replacement device. If a second SSD fails during this perioddata in the array is lost. Rebuild times for SSDs are typically quickerthan HDDs since they typically have less storage and are faster, howevertypically the rebuild can take the better part of an hour forcompletion.

A model of a failure of an SSD in the array 210 may be determined andused to improve the storage system 200 to mitigate a possibility of asecond storage disk failing while a first storage disk is beingreplaced. FIG. 3 illustrates a unit normal plot for a normaldistribution highlighting the probability of occurrence of the secondstorage disk failing in this manner. The Figure identifies a first,second, and third order sigma variation around a mean (μ), using anormal distribution for modeling the wearout. The failure model of anSSD uses the mean of the maximum number of writes as is shown in FIG. 4.The reliability of the SSD can be considered as a time to failureproblem. For example, assume that the SSD is a 256 GigaByte SSD and thatthe maximum writes on average per NAND flash device is 100,000. Further,assume that the wear leveling is near perfect then the writes will bespread evenly across all 256 GigaBytes so the total writes (the maximumwrites) will be given byWRITES_(TOTAL)=(256×10⁹)×(1×10⁵)=256×10¹⁴=2.56×10¹⁶.

Thus, if the system is writing a 4 kilobyte (KB) block, if the storagesystem 200 is to have, on average, a MTTF of 5 device-years, the desiredIOPs can be calculated as: IOPS_(WRITE)=(2.56×10¹⁶)/(3.16×10⁷sec/year)=162 megabytes(MB)/sec.

In an example, the 5 years may be a desired MTTF for a relatively highaverage IOPs application environment. In another example, the storagesystem 200 may be desired to have a MTTF of 6 device-years, or 8device-years. The MTTF may be a predetermined value that may be used todetermine an efficient manner to store the data across the array ofstorage disks 220, and minimize the possibility of the second storagedisk failing while the first storage disk is being replaced.

However, in the above example of using SSDs in the RAID-5 scheme, theSSDs will all continue to wearout in a relatively small time window sothe probability of the second disk failing while the first disk is beingreplaced is relatively high. For example, FIG. 4 and FIG. 5 illustratethat the maximum writes failure distribution of FIG. 4 can be mappedinto a time to failure distribution in FIG. 5. Further, FIG. 6 shows amapping of one distribution to another.

Returning now to the current example, where the SSD has a MTTF of 5years and a narrow failure window of, say a first order sigma of 4 weekson either side of the mean. Typically, the NAND flash maximum writesoccur in a much narrower window than the example. Further, as the IOPsincrease, especially for environments with lots of writes, the windownarrows even further. For example, the first order sigma may be 1 weekon either side of the mean in other examples. Other examples, dependingon the number of writes, may have different deviation values.

In the current example, the storage system 200 has an 8 week windowwhere roughly 68% of the SSD wearout failures are predicted to occur.This is an 8 week window centered around a 5 year MTTF. Thus, assumingthat the first SSD wearout failure occurs at week 256, and that it takesup to a week to detect the failure, procure a replacement device,schedule the replacement and rebuild the drive, the probability ofencountering the second SSD wearout failure in the RAID-5 array beforethe repair is complete, and thus have a data loss, can be calculated.For example, the probability may be calculated using Markov Models, or Ztables of a normal distribution, or any other technique.

For example, using Z tables for a normal distribution the probability offailure in the next week from −1 sigma to −0.75 is 0.0679. Thus, thereliability for a single SSD from week 256 to 257 is 0.9321. Since, inthis example 9 drives are left in the array, the probability of any 1 ofthese failing in the next week determines the probability of the secondSSD failing while the first SSD is being replaced. Differentpermutations of device failure may be ignored, since any 1 additionalfailure or combination of failures results in a data loss. Therefore,calculating the probability that all the SSDs survive and subtractingthat from 1 results in F_(RAID-5) (256<t<257)=1−[R1*R2*R3 . . .R9]=1−R⁹=1−(0.9321)⁹=0.469.

In other words in the example of a 10 device SSD RAID-5 array where afirst drive fails early in the wearout period at week 256, the storagesystem 200 has roughly a 50/50 chance of hitting a second SSD failureand thus having a data loss event if it take up to a week to effect therepair. Since this is the first SSD failure in the 10 SSD array and forthe normal distribution assumed in this example the bulk of the other 9devices are predicted to fail in an 8-week window too. Thus, RAID-5provides little if any real protection at all against data loss in thisscenario. Further, if the first order sigma window is reduced from 8weeks to 2 weeks it would be far worse than the example describedherein.

In another example, consider that the controller 210 implements a RAID-6scheme for the array of storage disks 220 that includes SSDs. In theRAID-6 scheme, data loss does not occur until three SSDs fail. UtilizingRAID-6 with the same 1 week repair and 8 week±1 sigma window, theprobability calculation of a data loss includes estimating theprobability of having 2 additional SSD fail during the up to 1 weekperiod to execute the repair. The failure permutations are as follows:

$\begin{matrix}{{F_{{RAID} - 6}\left( {256 < t < 257} \right)} = {\left\lbrack {F\; 1*\left\lbrack {1 - \left\lbrack {R\; 2*R\; 3*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 8} \right\rbrack} \right\rbrack} \right\rbrack +}} \\{{{\left\lbrack {F\; 2*\left\lbrack {1 - \left\lbrack {R\; 1*R\; 3*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 8} \right\rbrack}\; \right\rbrack} \right\rbrack \mspace{11mu} \ldots} +}} \\{\left\lbrack {F\; 8*\left\lbrack {1 - \left\lbrack {R\; 1*R\; 3*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 7} \right\rbrack} \right\rbrack} \right\rbrack} \\{= {8*\left\lbrack {F\; 1*\left\lbrack {1 - \left\lbrack {R\; 2*R\; 3*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 8} \right\rbrack} \right\rbrack} \right\rbrack}} \\{= {8*\left\lbrack {\left\lbrack {1 - {R\; 1}} \right\rbrack*\left\lbrack {1 - \left\lbrack {R\; 2*R\; 4*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 8} \right\rbrack} \right\rbrack} \right\rbrack}} \\{= {8*\left\lbrack {\left\lbrack {1 - R} \right\rbrack*\left\lbrack {1 - {R\; 7}} \right\rbrack} \right\rbrack}} \\{= {8*\left\lbrack {\left\lbrack {1 - 0.9321} \right\rbrack*\left\lbrack {1 - {(0.9321)7}} \right\rbrack} \right.}} \\{= {0.2112.}}\end{matrix}$

Thus, even with RAID-6, on average one out of every 5 times one attemptsto replace a failed SSD, 2 additional drives will wearout before therepair can complete and data loss will occur. Again, since this isprobability calculation is for the first SSD replacement for the 10 SSDarray and the bulk of the remaining 8 drives will fail in this next 8week window, the data loss will be a common occurrence even with a morerobust RAID-6 protection.

In a more extreme scenario, so as to do everything possible to make RAIDwork effectively for SSDs, assume that the repair window is no more thana single day (this includes, having a 24×7 coverage of the storagesystem 200 and stocking replacement SSDs on site so that the systemdetects and replaces a failed drive). Even in such an extreme scenario,for RAID-5:

F _(RAID-5)(256<t<256.035)=1−[R1*R2*R3 . . . R9]=1−R9=1−(0.9887)9=0.097.

Now, on average the storage system 200 encounters a data loss roughly 1out of every 10 attempts of replacing a storage disk. Thus, since thefirst of 10 SSD that are expected to wearout during the 8 week period,the possibility of data loss is not acceptable.

In case of RAID-6:

$\begin{matrix}{{F_{{RAID} - 6}\left( {256 < t < 256.035} \right)} = {\left\lbrack {F\; 1*\left\lbrack {1 - \left\lbrack {R\; 2*R\; 3*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 8} \right\rbrack} \right\rbrack} \right\rbrack +}} \\{{{\left\lbrack {F\; 2*\left\lbrack {1 - \left\lbrack {R\; 1*R\; 3*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 8} \right\rbrack}\; \right\rbrack} \right\rbrack \mspace{11mu} \ldots} +}} \\{\left\lbrack {F\; 8*\left\lbrack {1 - \left\lbrack {R\; 1*R\; 3*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 7} \right\rbrack} \right\rbrack} \right\rbrack} \\{= {8*\left\lbrack {F\; 1*\left\lbrack {1 - \left\lbrack {R\; 2*R\; 3*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 8} \right\rbrack} \right\rbrack} \right\rbrack}} \\{= {8*\left\lbrack {\left\lbrack {1 - {R\; 1}} \right\rbrack*\left\lbrack {1 - \left\lbrack {R\; 2*R\; 4*R\; 4\mspace{14mu} \ldots \mspace{14mu} R\; 8} \right\rbrack} \right\rbrack} \right\rbrack}} \\{= {8*\left\lbrack {\left\lbrack {1 - R}\; \right\rbrack*\left\lbrack {1 - {R\; 7}} \right\rbrack} \right\rbrack}} \\{= {8*\left\lbrack {\left\lbrack {1 - 0.9887} \right\rbrack*\left\lbrack {1 - {(0.9887)7}} \right\rbrack} \right.}} \\{= {0.0069.}}\end{matrix}$

In this case, on average 7 of every 1000 repair attempts produces dataloss. Nevertheless, if the wearout is quicker, say over a 6 week or 4week period instead of 8 or if the storage system 200 is unable toreplace the failed drive within 1 day the probability of encountering adata loss event quickly increases. Further, repairing a failed diskwithin 1 day is atypical, and rather ambitious. Hence, if the repairtime is extended for any reason the probability of secondary failuresagain increases.

Thus, the traditional RAID schemes are rendered ineffective in thestorage system 200 that uses the array of storage disks 220 thatincludes storage devices subject to wearout failure (such as SSDs), andwhere the storage system 200 employs techniques to recover failure of astorage disk by accessing data on other storage devices. For example,the wear leveling technology used on SSDs tends to make the distributionof writes to specific address of the drives irrelevant since wearleveling itself, by design, balances writes equally across all thestorage elements (such as across NAND flash used in the SSDs). The mostprevalent failure mode for SSDs subject to high numbers of writes, acommon usage scenario, is wearout failure as the write limit is reachedand thus SSDs when used in RAID arrays will tend to have clusters ofdevice failure at or around the same time. This makes it highly likelyto encounter a secondary SSD failure before the first SSD failure can berepaired and data restored to the RAID array, thus resulting in dataloss.

The technical solutions described herein utilize different techniques todistribute storage disk failures due to wearout over a longer period oftime so that the probability of encountering a second disk failureduring the repair of a first is reduced, thus rendering RAID aneffective option for protecting data on the storage disks, such as SSDs.

Variation of Checksum Writes

In one embodiment, the technical solutions utilize a RAID structure inwhich the writes to the storage disks are not equally distributed acrossall the storage disks in the array 200 (which is a goal of HDD basedRAID arrays to optimize performance). RAID-3, for example, with a singleparity drive (all parity resides on this one drive), is viewed asdeficient compared to RAID-5 for HDDs, since RAID 3 does not spread theparity field evenly across all the drives as is done with RAID-5. Theexamples described herein distribute parity structure, such as inRAID-5, but not using an equally balanced structure. Instead thedistribution of parity field on each device in the array is controlledand different so that the writes to each device over time is different.The number of parity fields on each storage disk is different, spreadingthem across all the storage disks. Thus, the RMW (Read Modify Write)operations that write data across the storage disks, update (rewrite)the parity when data is written to any storage disk. The storage diskwith more parity field (larger size) will experience more writes overtime. Thus, by distributing parity differently, the storage system 200alters the time to wearout for the storage disks. The more parity storedon the storage disk, the sooner that storage disk will wearout. Thestorage system 200 spreads the parity so that the ±1 sigma distributionof wearout failures does not intersect with the other storage disks. Inaddition, the technical solutions described herein facilitate using thestorage disks longer and avoiding replacing the disks prematurely,rather only when the storage disks have wornout.

The examples of the technical solutions are described using SSDs asstorage disks, however, the techniques described are applicable not onlySSDs, but to other storage devices, such as flash cards, flash PCIcards, clash SAS Cards, flash DIMMs, and other storage devices subjectto wearout failure, and those that employ recovery techniques whichrequire access to data on other storage devices.

FIG. 7 illustrates example logic to reduce the occurrence of data lossin the storage system 200. The controller 210 may implement the logic,for example by executing computer instructions that represent the logic.The controller 210, by implementing the logic, varies a number of parityfields per storage disk in the array 210 so that the wearout periodsassociated with the respective storage disks in the array 210 aredistinct and do not occur at the same time. The controller 210 of thestorage system 200 receives a request to write new data to the array ofstorage disks 220, as shown at block 705. The storage system 200 may usea read modified write (RMW) procedure to write data across the storagedisks. For the RMW procedure, the controller 210 reads a current parityand a current data that would be updated with the new data, as shown atblock 710. The current parity and current data is stored across thestorage disks in the array of storage disks 220. The controller 210generates new parity corresponding to the new data, such as by an XOR ofthe current data and parity with the new data, as shown at block 715.The above process may be represented formulaically as below.

Read: DATA_(current) for storage disk to be updated

Read: PARITY_(current)

Generate: PARITY_(new)=DATA_(current)+DATA_(new)+PARITY_(current)

Write: DATA_(new) for SSD to be Updated

Write: PARITY_(new)

The new data and the new parity is then written across the storage disksaccording to the recovery scheme implemented, as shown by blocks 720 and725. RMW is more efficient than reading and XORing all the data acrossall the storage disks, except the one to be updated with the new data togenerate the new parity. The RMW operation results in writes to 2storage disks, one for the new data and one for the new parity writtento each storage device. Thus, the RMW operation writes the new parityevery time there is new data written to a storage disk in the array 220.Returning to the previous example of a RAID-5 array with 10 SSDs andassuming the writes are predominately RMWs, there is a ratio of 9 to 1parity writes to data writes. Accordingly, for a SSD with a MTTF of 5years (260 weeks) following a normal wearout distribution with a firstorder sigma of 4 weeks, shifting writes on each drive by a multiple of 8weeks achieves staggering the wearout so that the probability of asecond device failing during a repair is acceptable, such as above apredetermined threshold. Accordingly, the controller 210 stores the newparity across the storage disks based on a stagger schedule, as shown atblock 725. The stagger schedule indicates, to the controller 210, how tostagger write operations across the storage disks in the array 220. Inan example, the stagger schedule indicates how to stagger writeoperations of the checksum data, such as parity or RS codes, across thestorage disks.

FIG. 8 illustrates how much to stagger the parity fields on the 10storage disks in the above example to avoid any overlap in the wearoutperiods for each storage disks. Since there are 10 storage disks and thebulk of the wearout occurs ±1 sigma around a MTTF or 260 weeks movingthe storage disk with the highest number of parity fields out to roughly40 weeks results in a calculation such as below.

For an odd number of disks—

${{Weeks}\mspace{14mu} {prior}\mspace{14mu} {to}\mspace{14mu} {Mean}} = {{\frac{N}{2} \times 2\sigma} = {{\frac{10}{2} \times 2(4)} = 40}}$

For an even number of disks—

${{Weeks}\mspace{14mu} {prior}\mspace{14mu} {to}\mspace{14mu} {Mean}} = {{\left\lbrack {\frac{N}{2} \times 2\sigma} \right\rbrack - \sigma} = {{{\frac{10}{2} \times 2(4)} - 4} = {{40 - 4} = 36}}}$

Accordingly, the stagger schedule is generated to stagger the writes onthe storage disks by staggering the parity fields on each drive. Thestaggering spreads the time to wearout so that each storage disks wearsout at a different time, and prevents any significant overlap betweenfailure time of one storage disk with replacement time of anotherstorage disk. FIG. 9 illustrates example logic to generate the staggerschedule and FIG. 10 illustrates an example stagger schedule generatedfor the 10 storage disk example being discussed herein. In an example,the controller 210 generates the stagger schedule. Alternatively, thecontroller 210 receives the stagger schedule and uses the staggerschedule.

The generation of the stagger schedule includes determining a writeratio of the parity to data according to the number of storage disks inthe storage system 200 as well as the recovery scheme being implemented,as shown at block 905. In the example storage system with 10 storagedisks with a RAID-5 scheme, there is a 9 to 1 write ratio of parity todata with the RMW updates. The stagger schedule is based on a value ofan average writes per week to reach the write maximum for the storagedisks in the system, which is calculated, as shown at block 910. In theexample, to get to the 2.56×10¹⁶ write maximum, the calculation resultsin 9.84×10¹³ average writes per week. In other words, the week 260, thenormal MTTF, is used as the center point to adjust the writes across thestorage disks. That is the stagger schedule is initialized according toa 1.0 (260/260) ratio of weeks to wearout, and then the writes for eachstorage disk are calculated and adjusted. An amount of additional writesfor a first storage disk to reach the wearout an order of deviationearlier than MTTF is calculated, as shown at block 915. Thus, in theexample in order to reach the maximum writes 36 weeks early or at 224weeks (260−36) from time zero a first storage disk takes 1.16 (260/224)more writes per week on average. Although the example uses a first orderdeviation, a second, or a third order deviation may be used in otherexamples. So that a storage disk does not fail while another storagedisk is being replaced upon failure, the wearout of the storage disksare spread from the first order deviation before the MTTF and the firstorder deviation past the MTTF. Therefore, the wearout of a final storagedisk is delayed until the first order deviation past the MTTF, as shownat block 920. In the ongoing example, the first order deviation 36 weekspast the MTTF 260 is week 296. Accordingly, the controller 210 reduces aratio of writes for a tenth storage disk in the array by 260/296 or0.878 writes. Write ratios for each of the storage disk in the array 220is calculated in this manner.

Column 5 in FIG. 10 illustrates the adjustment ratio for all the storagedisks in the array 220 of the example. The write ratios thus calculatedare standardized, as shown at block 925. For example, the write ratiosmay be standardized per 1000. Column 6 illustrates the relative numberof writes relative 1000 write for a normalized storage disk with thestandard MTTF of 260 weeks per the example. Using the standardized writeratios, a size of parity portions for each storage disk is determined,as shown at block 930. Since, in the example, the parity write to dataratio is 9 to 1, by dividing the total writes per 1000 in Column 6 by 9and rounding to the nearest whole number, the relative number of parityfields on each storage disk is calculated. Results for the example areillustrated in column 7. The relative number of parity fields on eachstorage disk indicate a size of a portion (i.e. portion size) of theparity data to be written to each storage disk in order to produce thewrite differences in column 6. The calculations may be rounded toproduce integers. For example, since dividing by 9 does not alwaysproduce integers the results may be round which may deviate the writeratio by a negligible amount, that does not produce any significantdifference in the mean-time to wearout skew, as can be seen bycalculated the actual writes difference in Column 8. FIG. 10 identifiesthe values of MTTF, and σ used to determine the write ratios and thecorresponding parity portion sizes for the 10 storage disk array in theexample. Other examples may use different MTTF and/or σ values tostagger the parity portions, and consequently the writes to the storagedisks differently.

The controller 210 assigns the storage disks in the array 220 a portionsize of the parity according to the corresponding MTTFs based on thestagger schedule generated, which is used to write a variable size ofthe parity in each of storage disks, as shown at block 725. For example,a first storage disk in the array 220, that has the shorted estimatedmean time to wearout gets the greatest number of parity fields. In anexample, if the storage disks in the array 220 are all at a similarstage relative to their wearout, such as the storage disks wereinstalled at the same time, the storage disks are assigned a MTTF andsubsequently, a corresponding parity portion size is assigned to eachrespective storage disk. In the ongoing example the first storage diskwith the shortest MTTF has a write ratio of 129 to 98 parity fields ascompared to the storage disk with the longest estimated MTTF. The 129 isonly about 30% higher writes than the 98 which is fairly negligible.

Thus, using the stagger schedule, the controller distributes the wearoutof the storage disks over a ±36 week period centered around the MTTF of260 weeks. Further, the stagger schedule facilitates that storage disksin the storage system 200 are not replaced prematurely, but only whenthe write limit of each storage disk has been reached.

In yet another embodiment, the storage system 200 employs a RAID-6 or aRAID-6 like recovery scheme, where two or more parity or checksum fieldsare utilized. The number of writes to each storage disk is madedifferent for each device by changing the number of parity or checksumfields on each device using logic similar to the example logic describedherein in context of the RAID-5 or RAID-5 like recovery scheme. As canbe understood by a person skilled in the art, in the embodimentsdescribed herein, tracking which stripes have the parity fields on whichdevices can be chosen in several different ways without affecting theresults of the technical solutions described. In an example, the stripesare chosen by allocating a specific block of addresses to each storagedisk. In another example, to gain additional granularity, the controller210 may assign a flag for each strip specifying which storage devicecontained parity.

The technical solutions described herein, thus facilitate a storagesystem to prevent data loss for multiple storage devices that tend toall fail at about the same time by reducing the probability ofencountering a secondary storage device failure (and hence the dataloss) before a first failed storage device can be replaced andreintegrated back into the RAID array. The multiple storage devices tendto fail at the same type since the storage devices fail due to a wearouttype failure after a predetermined number of maximum writes to thestorage devices. The technical solutions may be used irrespective of arecovery scheme being used by the storage system, such as RAID-5,RAID-6, or any other recovery scheme. The technical solutions reduce theprobability of the data loss by generating and/or using a staggerschedule. The stagger schedule is based on the maximum number of writesto the storage device, which is a predetermined value. The maximumwrites maybe a known value or an estimated. Further, the schedule isbased on a predetermined MTTF of the storage devices, which may be knownor estimated. The stagger schedule varies a time to failure of thestorage disks by controlling write operations to each storage diskconsequently spreading the time to failure of the storage disks overtime. The time to failure is varied across the storage disks in thearray by varying the number of parity fields vs. data fields on eachstorage disk in the data stripe. Thus, the number of writes over time isdifferent for each storage device and hence the wearout time of eachstorage device will be different and be spread over time.

Spare Disk

In a second embodiment, the technical solutions use an additionalstorage disk to distribute the storage disk failures due to wearout overa longer period of time so that the probability of encountering asecondary failure during the repair of a past error is reduced. Thetechnical solutions using the additional storage disk thus, renderrecovery schemes, such as RAID an effective option for protecting dataon Flash-based storage disks that are susceptible to wearout failures.

For example, a spare storage disk is added to the array of storagedisks. The spare storage disk temporarily replaces each drive in thearray with the spare for specific periods of time so that the wearouttime for each storage disk is varied. The performance overheadassociated with copying data from a storage disk to the spare storagedisk, using the spare for the specific period of time, and then copyingthe data back to the original storage disk, is overcome by performingsuch operations at a lower priority tasks that do not impact overallsystem performance. Further, the controller 210 avoids the performanceoverhead by initiating the operations at least prior to a predeterminedtime from an estimated wearout of a storage disk in the array 220.

FIG. 11 illustrates the storage system 200 with a spare storage disk1110 that in communication with the controller 210 like any otherstorage disk from the array of storage disks 220. The spare storage disk1110 may be of the same type as any of the storage disks in the array220, or alternatively of another type.

In an example, the maximum writes associated with wearout with thestorage disks in the array 220 are varied across the storage disks inthe array 220 without having to adjust the number of parity fields oneach storage disk. Alternatively, the maximum writes are varied usingthe spare disk 1110 in conjunction with varying the parity fields oneach storage disk in the array 220.

Embodiments of the present invention may be a system, a method, and/or acomputer program product at any possible technical detail level ofintegration. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects ofembodiments of the present invention. Returning to the example system of10 storage disks RAID array, the spare storage disk 1110 is an 11thstorage disk. The description herein continues to use the exemplarycharacteristics of MTTF=260 weeks, and normal wearout distribution withfirst order sigma at 4 weeks.

Since in case of storage disk such as an SSD, the storage disk wears outwhen the maximum writes to the storage disk are exceeded, it is not atime to failure distribution. The estimated storage disk MTTF is notconstant, but varies based on how many writes occur over time, which mayvary. By tracking average writes per unit time interval a Time toFailure estimate is made and adjusted over time if the average writesper unit time interval varies due to changing application load. Thetechnical solutions described herein facilitate a dynamic monitoring ofaverage writes over time so that the a smaller time window to conductthe replacements is used to reduce replacing storage disks in the array220 before most of the useful life of the storage disk has beenexhausted. The technical solutions further provide a guided maintenancefor replacing the storage disks in the array 220, such as SSDs in a RAIDarray.

FIG. 12 is a flowchart of an example logic to shift the wearouts of thestorage disks in the array 220 using the spare storage disk 1110. Theexample logic may be implemented by the controller 210 according tocomputer readable instructions that may be stored on a memory. Thewearout shift process is initiated by using the spare storage disk 110to replace a first storage disk, as shown at block 1205 and 1210. Thefirst storage disk may be any storage disk in the array. Replacing astorage disk in the array 220 by the spare storage disk 1110 entails thecontroller 210 redirecting a read/write operation that is directed tothe storage disk to the spare storage disk 1110. Thus, for a writeoperation directed to the first storage disk, the controller 210performs the write operation on the spare storage disk 1110, as long asthe spare storage disk 1110 is replacing the first storage disk.

The spare storage disk 1110 replaces a storage disk in the array for aspecific duration, as shown by block 1215. At the end of the replacementduration the controller 210 copies the data from the spare storage disk1110 to the storage disk that was replaced, as shown at block 1220. Thereplacement duration is based on a predetermined deviation from theMTTF. For example, the predetermined deviation may be a first order, asecond order or the like. Accordingly, the replacement duration mayvaries. The controller 210 subsequently uses the spare storage disk 1110to replace a next storage disk in the array, as shown at block 1225. Thecontroller 210, in an example updates the replacement duration for whichthe next storage disk is replaced by the spare storage disk 1110. Forexample, the replacement duration may be a multiple of the initialreplacement duration, the multiplier corresponding to the iteration ofthe process. For example, the replacement duration for a second storagedisk may be twice that of the first storage disk, the replacementduration for a third storage disk may be thrice that of the firststorage disk, and so on. Other multiplying factors may be used to updatethe replacement duration. The controller 210 continues the process untilall the storage disks in the array 220 are replaced in this manner afterwhich, the controller 210 resets the replacement duration and therestarts the replacement process, as shown at block 1230.

FIG. 13 illustrates result of using the spare storage disk 1110 asdescribed in the specific example of 10 storage disks, and using aninitial replacement duration of 8 weeks (±4 weeks according to firstorder sigma). In the example, the controller 210 delays the wearout ofthe first storage disk by 8 weeks shifting the MTTF from week 260 toweek 268. The controller 210, at the end of 8 weeks copies data from thespare storage disk 1110 back to the first storage disk and uses thefirst storage disk again. The controller 210 proceeds to copy data froma next storage disk to the spare storage disk 1110 and use the sparestorage disk 1110 as the next storage disk for 16 weeks (replacementduration multiplied by iterator). This delays the wearout of a secondstorage disk by 16 weeks shifting the MTTF from week 260 to week 276. Atthe end of the 24^(th) week (16 weeks of replacement), the controllercopies data from the spare storage disk 1110 to the second storage diskand begins using the second storage disk again. The results ofcontinuing this process is illustrated in FIG. 13.

According to this example, shifting according to the ±4 week 1 sigma (8week primary wearout window) and updating the replacement duration everyiteration according to the iterator takes 440 weeks to complete for allthe 10 storage disks. Since the MTTF is estimated at week 260, if thereplacement process does not complete by the MTTF, the storage system200 continues to experience storage disk failures clustered at week 260.As can be seen from FIG. 13, at week 260, the controller 210 has justbegun to shift the wearout of storage disk-7 in the 10-disk array andnot yet done anything with the storage disks 8 and 9. The total numberof weeks to complete the wearout shifting with a single spare storagedisk as described is determinable as

${{Total}\mspace{14mu} {Weeks}} = {\sum\limits_{i = 1}^{N}({ixW})}$

where N is the total number of storage disks in the array 220 and W isthe initial replacement duration.

Accordingly, in the specific example with N=10, and W=8, Total Weeks=440weeks (˜8.5 years). Since the MTTF for wearout, in the exemplaryscenario is 5 years the controller 210 only completes the wearoutshifting for 7 of the 10 storage disks. Accordingly, if the controlleruses a different replacement period (W) and/or a different multiplier(i), the controller 210 may be able to complete the shifting of thewearout across all the storage disks. For example, instead of the full 8weeks, the controller 210 may use a narrower wearout window, say the ±2week 1 sigma (4 week wearout window) or the ±1 week 1 sigma (2 weekwearout window). For example, using W=4, Total Weeks=220 weeks, which isless than the MTTF of 260 weeks. FIG. 14 illustrates an exemplaryshifting schedule in such a case. Alternatively, the controller 210 maycontinue to use the replacement duration as 8 weeks in case the MTTF forwearout is greater than the resulting total weeks, say MTTF=9 years vs.5 years or the number of SSDs in the storage array is less. Thus, thecontroller 210, or alternatively a user of the storage system 200 mayconfigure the parameters of the replacement according to the MTTF,number of storage disks in the array 220, and the σ to be used to shiftthe wearout of the storage disks.

In another example, the storage system 200 uses a plurality of sparestorage disks and continues to use the replacement duration according tothe conservative larger wearout window. FIG. 15 illustrates an examplelogic for shifting the wearout using two spare storage disks. Thereplacement proceeds similar to the replacement using a single sparestorage disk described herein, however, two storage disks in the array220 are replaced by each of the spare storage disks respectively. Forexample, the controller 210 uses a first spare storage disk forreplacing the even numbered storage disks and a second spare storagedisk for replacing the odd numbered storage disks in the array 220.

For example, as shown in FIG. 15, the controller 210 initializes X=0 anduses the first spare storage disk to replace a storage disk-X in thearray 220, as shown at 1505A and 1510A. The storage disk-X is replaceduntil the replacement duration expires, after which data from the firstspare storage disk is copied back to the storage disk-X, as shown atblocks 1515A and 1520A. The controller 210 subsequently updates the X byincrementing it by 2 to replace the next even numbered storage disk, asshown at block 1525A. The controller 210 also updates the replacementduration corresponding to the next even numbered storage disk, as shownat 1525A. The controller 210 may continue this process for all the evennumbered storage disks in the array 220, as shown at block 1530A.

Additionally, as shown in FIG. 15, the controller 210 initializes Y=1and uses the second spare storage disk to replace a storage disk-Y inthe array 220, as shown at 1505B and 1510B. The storage disk-Y isreplaced until the replacement duration expires, after which data fromthe second spare storage disk is copied back to the storage disk-Y, asshown at blocks 1515B and 1520B. The controller 210 subsequently updatesthe iterator Y by incrementing it by 2 to replace the next odd numberedstorage disk, as shown at block 1525B. The controller 210 also updatesthe replacement duration corresponding to the next odd numbered storagedisk, as shown at 1525B. The controller 210 may continue this processfor all the odd numbered storage disks in the array 220, as shown atblock 1530B. The controller 210 may replace the even and theodd-numbered storage disks in parallel in an example.

In the previous exemplary scenario with 10-storage disk array withMTTF=260 weeks, and σ=±4 weeks, the two spare storage disks proceeds asdescribed further. FIG. 16 illustrates a schedule of the wearoutshifting in such case. Referring to the FIG. 16, the controller 210starts the wearout shift process by running the SPARE0 instead ofstorage disk-0 for the first 8 weeks and then copy the data from theSPARE0 to the storage disk-0 at that time. This delays the wearout ofthe storage disk-0 by 8 weeks shifting the MTTF from week 260 to week268. The controller additionally uses SPARE1 instead of storage disk-1for the first 16 weeks and then copies the data from the SPARE1 tostorage disk-1 at that time. This delays the wearout of the storagedisk-1 by 16 weeks shifting the MTTF from week 260 to week 276. At theend of the first 8 weeks the controller 210 copies data from the SPARE0to the storage disk-0 and begins using the storage disk-0 again. Thecontroller 210, at this time, copies the data from storage disk-2 to theSPARE0 and use the SPARE0 as the storage disk-2 for 24 weeks. Thisdelays the wearout of the storage disk-2 by 24 weeks shifting the MTTFfrom week 260 to week 284. The controller 210 then copies data from theSPARE0 to storage disk-2 and begins using storage disk-2 again. Thecontroller 210 continues this process for the even numbered storagedisks, storage disk-4, storage disk-6, and storage disk-8. The oddnumbered storage disks are handled in a similar manner.

The total weeks to complete the wearout time shift for the odd numberand even number storage disks in the array can be calculated as follows.

Total Weeks=Max(Total Weeks Odd, Total Weeks Even), where

$\begin{matrix}{{{Total}\mspace{14mu} {Weeks}\mspace{14mu} {Odd}} = {\sum\limits_{{i = 1},{i\mspace{11mu} {odd}}}^{N - 1}({ixW})}} \\{{{Total}\mspace{14mu} {Weeks}\mspace{14mu} {Even}} = {\sum\limits_{{i = 0},{i\mspace{11mu} {even}}}^{N}({ixW})}}\end{matrix}$ W = Initial  Replacement  Duration, andN = Total  Number  of  Storage  Disks

Accordingly, in the exemplary scenario, with N=10 and W=8, Total WeeksOdd=200 weeks, and Total Weeks Even=240 weeks. Thus, Total Weeks=240weeks. Hence, with two spare storage disks the controller successfullyshifts the wearout across the 10 storage disks by week 240, which is 20weeks prior to the MTTF of 260 weeks for wearout.

FIG. 17 shows the MTTF shift for each storage disk once the shiftingprocess completes. In an example, the controller may not shift thewearout for the last storage disk, which takes the longest to shift. Inthe specific scenario, the controller 210 skips the wearout shifting ofstorage disk-9, since it is the only one that still has its MTTF at week260. Skipping the shifting for the last storage disk reduces the totalweeks of the shifting process. For example, in the 10-disk scenario ifstorage disk-9 is not shifting, the time to complete the shifting isreduced from 240 to 200 weeks. Skipping the shifting of the last disk inthe example with a single spare storage disk reduces the time from 440weeks to 360 weeks (˜8.5 years to ˜6.9 years).

In addition to shifting the wearout across the storage disks in thearray, the technical solutions described herein prevent storage disksbeing replaced prematurely, but only when the full write limit of eachstorage disk has been reached.

The technical solutions described herein may be deployed as a softwareor firmware for the controller 210 of the storage system 200 that hasalready been deployed. In other words if a storage systems has beenshipped 2 years earlier, a computer product comprising computer readableinstructions, or alternatively a replacement controller may be added tothe system to prevent cluster wearout failure of the storage disks.

The technical solutions described herein thus facilitate a storagesystem to prevent data loss for storage disks that tend to all fail atabout the same time by reducing probability of encountering a secondarystorage disk failure (and hence data loss) before a first failed storagedisk can be replaced and reintegrated back into an array of the storagesystem, such as a RAID array. The technical solution reduces theprobability of the data loss by replacing storage disks in the arraysequentially. The storage disks may be replaced individually atdifferent times, or in parallel. Replacing the disk varies the number ofwrites across the storage disks in the array, and hence shifts thewearout of the storage disks. Thus, the technical solutions describedherein vary the writes across the storage disks through the use of aspare storage device that systematically temporarily replaces eachstorage device in the array for differing amounts of time. The differingamounts of time spreads the wearout window for the storage disks overdifferent periods of time.

Disk Replacement Notification

According to yet another embodiment, the technical solutions facilitatemonitoring and tracking a number of writes to the storage disks and therate of writes to the storage disks in the array 220. As the maximumwrite limit for storage disk(s) in the array 220 draws nearer, thecontroller 210 may generate and trigger an alert so that anadministrator arranges for replacement(s) can be made prior to reachingwearout for the storage disks in the array 220. For example, thecontroller 210 may generate the alert would indicating that a firststorage disk in the array 220 is to be replaced. Once that repair iscomplete the controller 210 generates a second alert, after passage of apredetermined time, for a second storage disk in the array 220. Thecontroller 210 may continue to generate periodic alerts in this mannerover an extended period. The technical solutions described herein,further facilitate triggering an alert prior to the wearout period of astorage disk so that the storage disk can be replaced before theprobability of encountering a second device failure during the repairand rebuild increases substantially. Additional consideration isprovided so that the storage disk is not replaced grossly prematurely,before the wearout of the storage disk.

In addition, the technical solutions described herein facilitate animprovement of rebuilding and replacing a storage disk that is close towearout. For example, if the administrator replaces a storage device inresponse to the alert, the rebuild of the data from the storage diskthat is to be replaced to a new replacement disk includes reading thedata from the storage disk and writing the data to the replacement disk.The controller 210 permits reading data from the storage disk even ifthe write limit of the storage disk has been reached. Thus, thereplacement disk is installed without performing a data rebuild, whichis generally performed when replacing an already failed disk. The datarebuild generally includes, reading data from all the storage disks inthe array 220 and generating the data from the failed storage disk, suchas by using an XOR operation of the data and parity together toreproduce data from the failed storage disk.

In another example, the controller 210 monitors the number of writes andthe rate of writes over time to the storage disks in the array 220. Asthe maximum write limit for a first storage device draws nearer, thecontroller triggers an alert indicative to replace all the storage disksin the array 220 prior to reaching wearout for the first storage disk.The controller may generate such an alert closer to an expected wearoutof the first storage disk, compared to the alert in the earlier examplebecause the delay between replacing the storage disks in this mannerwould be lesser than replacing the storage disks sequentially. Forexample, the administrator replaces the first storage disk immediatelyfollowed by the next, and so on until all the storage disks in the array220 are replaced. Additionally, the alert in this manner facilitates theadministrator to order all the replacement storage disks in the array atthe same time and replacements are done as a single set of serialevents. Thus, repeated repair actions to the system are reduced, whichin turn save time and costs, especially if there is travel involved forthe administrator (or a technician) to replace the storage disks.

Typically, the storage system 200 initiates a Predictive Failure Alert(PFA) upon detecting wearout of one of the storage disks in the array220 of the storage system 200. The technical solutions facilitate thecontroller to provide a notification earlier than, and distinct to, thePFA so that a first storage disk is replaced before the wearout periodfor the first storage disk in the array 220 is reached. Thus, theprobability of encountering a secondary storage disk failure before thefirst storage disk is replaced and reintegrated back into the array 220is reduced. Accordingly, the storage disks in the array 220 remain inuse for the vast majority of their useful life. The notification isgenerated before wearout, before the storage disks reach the point wherea secondary storage disks failure are likely to occur. As a result, someamount of useful life of the storage disks is lost.

The storage system issues a first alert to replace storage disk(s) inthe array 220 according to the expressionT_(start)=N×[T_(repair)+T_(wait)]+T_(1-sigma),

where, T_(start)=Time prior to MTTF when the first alert is issued;T_(repair)=Time to detect the storage disk failure, procure areplacement storage disk, replace the storage disk & rebuild data so asto integrate the replacement storage disk into the array 220(T_(repair)=T_(detect)+T_(order)+T_(replace)+T_(rebuild));T_(wait)=Time to wait prior to generating a subsequent alert;T_(1-sigma)=Time that is first order σ from the MTTF; andN=Number of storage disks in the array.

For example, with respect to the exemplary RAID-5 system describedherein, which has 10 storage disks with a MTTF of 260 weeks (that is 5device years), the system will generate and issue a first alert asdescribed below. As a reminder, in the example system being considered,the T_(repair)=1 week, the T_(wait)=1 week; the T_(1-sigma)=4 weeks; andN=10. Thus, considering that repairing a storage disk takes up to a weekand 68% of the wearout is expect to occur in an 8 week window centeredaround an MTTF of 5 device years (260 weeks), the system determines whento generate and issue a first alert.

Accordingly,

$\begin{matrix}{T_{start} = {{N \times \left\lbrack {T_{repair} + T_{wait}} \right\rbrack} + T_{1 - {sigma}}}} \\{= {{10 \times \left\lbrack {{1\mspace{14mu} {week}} + {1\mspace{14mu} {week}}} \right\rbrack} + {4\mspace{14mu} {weeks}}}} \\{= {24\mspace{14mu} {{weeks}.}}}\end{matrix}$

Thus, the exemplary storage system generates and issues a first alert 24weeks prior to MTTF, that is at week 236 (260−24). Accordingly, at week236 the notification indicating that a first storage disk in the arrayis to be replaced. The storage system 200 may automatically order areplacement disk in response to the notification. Alternatively, asystem administrator orders the replacement disk. Consider that thefirst storage disk is replaced by the replacement disk within a week.The system then waits for 1 week (T_(wait)), after the replacement iscomplete, before issuing a next alert indicating that a second storagedisk in the array needs to be replaced. Thus, in the exemplary scenariothe second notification is generated and issued 22 weeks before the 5year MTTF point. Accordingly, in this example, the 2^(nd) replacementcompletes at 20 weeks prior to MTTF, the 3^(rd) replacement at 18 weeksprior to MTTF and so on as listed in Table 1.

TABLE 1 Storage Weeks Prior to Weeks from Time Disk MTTF Zero %Premature 0 24 236 9.23 1 22 238 8.46 2 20 240 7.69 3 18 242 6.92 4 16244 6.15 5 14 246 5.38 6 12 248 4.62 7 10 250 3.85 8 8 252 3.08 9 6 2542.31 5.77 average

Referring to FIG. 18, the storage system 200, in this exemplaryscenario, initiates the notification to replace storage disk(s) in thearray 220, starting 24 weeks before the 5 year MTTF, as illustrated bythe dashed line illustrated. This start time of 24 weeks is determinedby considering the number of storage devices in the array 220, therepair time, and the delay time. For example, the storage system 200issues a first alert at [(1+1)×10] weeks prior to the 4 week−1 (minus 1)sigma point on the distribution since there are 10 drives in the array,up to 1 week repair time, and 1 week delay before generating the nextalert. As illustrated in Table 1, this schedule facilitates the systemto replace all the storage disks in the array before the wearout periodis reached. The replacement of all the storage disks, as illustrated, iscompleted before the 4 week−1 sigma point (260−6=256 weeks), that isprior to the 5 year MTTF. In addition, as each subsequent storage diskis replaced, the probability of a wearout of any of the remaining drivesin the array that have yet to be replaced is reduced. Thus, the systemallays the storage disk data loss because of wearout associated with arecovery scheme, such as RAID-5. The tradeoff, in this particularexample, is a 5.77% loss of useful life time of the storage disks onaverage (see table 1).

The storage system 200, for example via the controller 210, utilizessmart data information from the storage disks along with data tracked bythe controller 210 itself to determine rate of the array approaching awearout. Since the wearout with storage disks, such as SSDs, isassociated with a number of writes, the wearout of a storage system 200is application environment dependent. Hence, the MTTF is dynamic. Thecontroller 210, by monitoring the rate of writes to the storage disks inthe array 220, determines an accurate estimate of the MTTF for theparticular storage system 200. For example, the controller 210determines a dynamic estimate of the MTTF by considering not just thetotal writes to the storage disks over time, but the rate of change ofwrites over time so that the estimated MTTF can be adjusted along withthe 1 sigma point of the wearout distribution.

In yet another example, the storage system generates and issues anotification signaling that all the storage disks in the array 220 areto be replaced so that replacement disks for all the storage disks areordered at one time. This facilitates the storage system to issue thenotification later than a notification time in case of sequentialreplacement as described elsewhere in this document. Thus, the storagesystem 200 reduces the duration of time prior to reaching MTTF for thealert to be generated, and consequently reduces loss of useful life ofthe storage disks. For example, the alert to replace all the storagedisks may be generated at a time T_(start-all), determined as:

T _(start-all) =T _(detect) +T _(order) +[N×[T _(replace) +T _(rebuild)+T _(wait) ]]+T _(1-sigma)

In the exemplary RAID-5 scenario, T_(start-all), determined according tothe above considerations, is—

$\begin{matrix}{T_{{start} - {all}} = {{1\mspace{14mu} {day}} + {1\mspace{14mu} {week}} + \left\lbrack {10 \times \left\lbrack {{1\mspace{14mu} {hour}} + {1\mspace{14mu} {hour}} + {22\mspace{14mu} {hours}}} \right\rbrack} \right\rbrack + {4\mspace{14mu} {weeks}}}} \\{= {{8\mspace{14mu} {days}} + \left\lbrack {\text{∼1}\mspace{14mu} {day}} \right\rbrack + {28\mspace{14mu} {days}}}} \\{= {\text{∼37}\mspace{14mu} {{{days}\left( {\text{5-6}\mspace{14mu} {weeks}} \right)}.}}}\end{matrix}$

Here, it is assumed that detection of a wearout takes 1 day, orderingthe replacement disks takes 1 week, replacing the disk takes 1 hour,rebuilding the data takes another 1 hour, and that the storage systemreplaces a subsequent storage disk at least one day after testing thecurrent replacement for almost a day.

FIG. 19 illustrates a flowchart of example logic to forestall data lossin the storage system 200, particularly data loss that is caused by asecondary wearout while a first storage disk of the array is beingreplaced due to wearout of the first storage disk. In an example, thecontroller 210 implements the logic, which may be part of a computerprogram product that contains computer executable instructions.According to the logic, the controller 210 monitors the rate of writesfor the storage disks in the array 220, as shown at block 1905. In anexample, the controller 210 monitors the rate of writes for each storagedisk separately. In another example, the controller 210 monitors therate of writes for selected storage disks in the array and determines arate of write for the overall array, such as by averaging the monitoredrate of writes. The controller 210 determines the MTTF for a storagedisk in the array based on the monitored rate of writes, as shown at1915. For example, the storage devices may have a predetermined maximumnumber of writes. The maximum number of writes may be predetermined by amanufacturer of the storage disk, or alternatively may be predeterminedby the controller 210. In an example, the controller 210 prohibits anywrites to a storage disk that has encountered the maximum number ofwrites. The controller 210 may permit reading data from the storage diskthat has encountered the maximum number of writes. The controller 210determines the MTTF for the storage disk based on the maximum number ofwrites and the rate of writes.

The controller 210 may be configured to forestall the data loss byreplacing the storage disks in the array sequentially or by replacingall the storage disks in the array in conjunction. In another example,the controller 210 may be configured to replace the storage disks in thearray in smaller batches of predetermined sizes. The flowchartillustrates the cases in which all the storage disks are replaced in asingle batch or all the storage disks are replaced in multiple batchesof 1 disk each. The controller 210 determines which is the case, asshown at block 1920. In case the controller 210 is configured to replaceall the storage disks in conjunction, the controller 210 determines astart time to issue a notification for replacement of all the storagedisks, as shown at 1940. For example, the start time may be determinedas T_(start-all) as described herein. The controller 210 issues thenotification to replace all the storage disks in the array at thedetermined start time, as shown at block 1942. The controller 210, in anexample, may issue an order for the number of replacement disks in thearray 220 (not shown). In another example, in response to thenotification, the administrator of the storage system may place theorder and initiate replacement of the storage disks once the replacementdisks are procured.

In case the controller 210 is configured to replace the storage diskssequentially, the controller 210 determines the start time to issue thenotification to replace each storage disk in the array, as shown atblock 1930. The controller 210, in an example, may generate the starttime corresponding to each storage disk sequentially, that is thecontroller 210 determines a first start time for replacement of a firststorage disk and after completion of replacement of the first storagedisk, the controller determines a second start time for a second storagedisk, and so on. Alternatively, the controller 210 generates a schedulethat includes all the start times for each of the storage disk in thearray 220. At the start time for each respective storage disk in thearray 220, the controller 210 issues a notification to replace therespective storage disk, as shown at 1932. The controller 210 waits fora predetermined duration (T_(wait)) until it issues a subsequentnotification for replacement of a next storage disk in that array 220,as shown at block 1936. If all notifications are complete the controller210 ends the notifications for the time being, and may resume monitoringthe rate of writes to iteratively continue the entire process, as shownat block 1938. Alternatively, if all the storage disks in the array 220have not yet been replaced, the controller 210 continues to issue thenotifications, as shown at 1932.

Since the wearout of a storage disk, such as SSD, occurs when themaximum writes to the storage disk are exceeded, the distribution ofreplacing the storage disks in the array is not modeled as a time tofailure distribution. The estimated storage disk MTTF is not constant,but varies based on how many writes occur over time which may vary. Bytracking average writes per unit time interval a MTTF estimate is madeand adjusted over time if the average writes per unit time intervalvaries due to changing application load. Thus, the storage system 200provides dynamic monitoring of average writes over time so that the asmaller time window is used to place the storage disks before most oftheir useful life has been exhausted by providing guided maintenance forreplacing the storage disks in the array 220. Such a guided maintenancemay be deployed as a computer product that contains control instructionsfor the controller 210 of the storage system 200 that has already beendeployed.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application, or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

1.-10. (canceled)
 11. A storage system comprising: a plurality ofstorage disks in an array; and a controller that stores data across thestorage disks, wherein the controller forestalls data loss caused bywearout of storage disks in the array of storage disks by the controllerbeing configured to: monitor a rate of writes for a first storage diskin the array; determine a mean time to failure of the first storage diskbased on the rate of write; determine a start time based on the meantime to failure, a number of storage disks in the array, and a time toreplace a storage disk in the array; and issue at the start time, anotification, the notification includes an alert to replace the firststorage disk, wherein replacement of the first storage disk at the starttime forestalls the data loss caused by a wearout of a second storagedisk of the array in conjunction with a wearout of the first storagedisk.
 12. The storage system of claim 11, wherein the time to replace astorage disk in the array includes a time to detect wearout of a storagedisk, procure a replacement storage disk, replace the storage disk withthe replacement disk, and rebuild the data in the storage systemaccording to a recovery scheme of the storage system.
 13. The storagesystem of claim 11, wherein the start time is a first start time, thenotification is a first notification, and the controller is furtherconfigured to: issue at a second start time, a second notification, thesecond notification including an alert to replace the second storagedisk, wherein replacing the second storage disk at the second start timeforestalls the data loss caused by a wearout of a third storage disk ofthe array in conjunction with a wearout of the second storage disk,wherein the second start time occurs after the first start time, and thesecond start time is determined based on the mean time to failure, thenumber of storage disks in the array, the time to replace a storage diskin the array, and a predetermined delay since replacement of the firststorage disk.
 14. The storage system of claim 11, wherein the controlleris further configured to: generate a schedule to issue notifications forreplacement of each of the storage disks in the array, wherein theschedule comprises a plurality of start times corresponding to eachrespective storage disk in the array, and wherein a duration betweensuccessive start times includes a predetermined wait time; and issue aplurality of notifications, each of which is issued at a respectivestart time from the schedule.
 15. The storage system of claim 11,wherein the array of storage disks is maintained as a Redundant Array ofIndependent Disks (RAID), and wherein a storage disk in the array is oneof a solid state drive, a Flash memory, or a Flash Dual In-line MemoryModule (DIMM), interfaced using any one of a Peripheral ComponentInterconnect (PCI) card, a Serial Attached Small Computer SystemInterface (SAS).
 16. The storage system of claim 15, wherein thenotification includes alerts to replace all the storage disks in thearray in conjunction.
 17. The storage system of claim 16, wherein thearray of storage disks is a redundant array of independent disks (RAID),and wherein a storage disk in the RAID is one of a solid state drive, aflash memory, or a flash dual in-line memory module (DIMM), interfacedusing any one of a peripheral component interconnect (PCI) card, aserial attached small computer system interface (SAS).
 18. A computerprogram product for forestalling data loss caused by wearout of storagedisks in an array of storage disks in a storage system, the computerprogram product comprising a computer readable storage medium havingprogram instructions embodied therewith, the program instructionsexecutable by a processor to cause the processor to: monitor a rate ofwrites for a first storage disk in the array; determine a mean time tofailure of the first storage disk based on the rate of write; determinea start time based on the mean time to failure, a number of storagedisks in the array, and a time to replace a storage disk in the array;and issue at the start time, a notification, the notification includesan alert to replace the first storage disk, wherein replacement of thefirst storage disk at the start time forestalls the data loss caused bya wearout of a second storage disk of the array in conjunction with awearout of the first storage disk.
 19. The computer product of claim 18,wherein the start time is a first start time, the notification is afirst notification, and the computer readable storage medium furthercomprises instructions to: issue at a second start time, a secondnotification, the second notification including an alert to replace thesecond storage disk, wherein replacing the second storage disk at thesecond start time forestalls the data loss caused by a wearout of athird storage disk of the array in conjunction with a wearout of thesecond storage disk, wherein the second start time occurs after thefirst start time, and the second start time is determined based on themean time to failure, the number of storage disks in the array, the timeto replace a storage disk in the array, and a predetermined delay sincereplacement of the first storage disk.
 20. The computer product of claim19, wherein the notification includes alerts to replace all the storagedisks in the array in conjunction.